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ABSTRACT 

A distinction is made between program evaluation and 
project evaluation. The former is used to determine tha effectiveness 
of a particular program or to compare programs. The latter is used in 
the initial st.ages of program development to define and monitor 
progress; to identify problems at an early stage; to record what is 
done and whyji aad to provide decision- making data. Test selection 
criteria ana procedures for establishing external and internal 
validity are defined, program evaluation consists of th^e steps: (1)/ 
defining objectives; (2) identifying infoxmation to document 
attainment of objectives; (3) selecting or developing instruments to 
collect this information; (4) establishing success criteria for these 
instruments; and (5) developing a management system for the 
administratipn and analysis of evaluation forms. Project evaluation 
consists of four steps, each^having its own, evaluation form. Steps 
include*' specif ying tasks; monitoring pr5gress toward completion; 
documenting completion; and^evaluating completed tasks. The project 
'evaluation approach follows Provus* Discrepancy Evaluation Model, A 
role distinction is made between project evaluator as manager of 
eictivities and the program evaluator as impartial plannii^g 
consultant. (CP) ^ . • 
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THE PROCESS OF EVALUATION 

I 

" s! 

I. Perspectl ye 

In evaluating a program two types of evaluation may he of concern 
to the evaluator: program and project evaluation. Progra^ evaluatloh is 
used to determine how effective a particular program Is or to determine 

■ r . ■ 

If one program is more effective than another. Project evaluation la used 
'to t)rovide means of defining and monitoring program progress; to asQuT^ 



identification of problems before they disrupt activities in other parts 
of the program; to maintain a record of what is in fact done and w^y; ^nd 



I 

to provide data for declslon-maklrig. ^ ' , '% 



Both types of evaluation are carried out through the uae of measurement,; 
tools. A measurement tool is a defined technique ot collecting data that 




ar^ needed to conduct the evaluation. *The measurement tools^ moa^t^a|miliar 
to everyone are examinations and questionnaires. However^ checklldta, 
interviews, and direct observation cjf individuals are also measurement tools. 

In choosing the specific measurement tool to use In projeqft or pjrogram 
evaluation^ consideration must be given to /the reliability^ Validity, 



•availability of normative dat^, and utility of ''the measurement tool. Speci- 

fic guidelines for review of instrum^nlis are published by tt^ American J? 

1 

Psychological Association. ^- . ' 

* ■ ' ^ 

• .Reliability refers to the extent to which the measureuient tool is 

consistent in measuring whatever it measures. The reliability of the mea- 



J 

is>cc 
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surement tool is>coffimonly reported as a correlation coefficient; - A correla- 
tion coefficient is a measure of the degree of relationship between two 

1 i * 

American Psychological Association. Standards f or*" Educational and Psy - 
chological Tests and Manuals . Washi^^on, D.C. APA*1966 . 
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sets of measures. The c^fficlent may ho In terms of an altemate-fo 



rm 



reliability correlation coe ffici ent : split-half rolinblilty correlation , 
coefficient; or a test-retest reliability correlation coefficient- An 
alternate-form rei^labllity correlation coefficient indicates the extent 
to which two forms the measurement tool are consistent in their results. 
A split-half reliability correlation coefficient indicates t^ie extet^ to 
which the results of on^-half of the measurement tool are consistent with 
the result^ of the other half. A test-retest reliability correlation 
coefficient indicates the extent to which results obtained by administer- 
Ing the same measurement tool on two different occasions are consistent. 

Ones use of the tool will determine which type of reliability 
estimate Is relevant to the case. It is essential to establish the reli- 
ability of tjie measurement tool used in the evaluation study. If the tool 
is not reliable, the results obtained will be dictated by chance factors. 

Reliability is a necessary but not sufficient condition for validity, 
also an essential requirement for the measurement tool. Validity refers 
to the extent to which the tool measures what it is intended to. The 
validity of the measurement tool may be established in terms ot content 
validity, construct validity, predictive validity, or face valid^^ty. ^/ 
Cpntent >5^alldity assesses the extent to, which the tool adequately samples 
outcomes of the Universe of behaviors it is intended to measure. Construct 
validity refers to the extent to which the tool yields an accurate descrlp- 
tion" of the psychological construct it is intended to describe. Predictive 
validity refers to the extent to which the tool accurately Indttates future 
success In J;he area for which It Is used as a predictor. Face validity 
refers to the acceptability of the tool by the examinee in terms of apparent 
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uses to which it is put. Depondlng upon the use of th(? tool, the impor- 
tance of establishing each type of validity will vary. 

The measurement tool used in the evaluation study should also have 
established data on results observed from using the tool on groups similar 
to the present group of interest. Without relevant normative data^ inter- 
pretation of results becomes difficult. 

Finally, the decision of which particular measurement tools are to 
be used in the evaluation study should be based on the utility of the tool 
in terras of money, time to collect the data, and any other practical needs 
relevant to the case. 

Evaluation studies, however, must not only concern themselves with 
the quality of the measuring tools,, but also the conditions under which 
the tools are to be used. The conditions of the study must be arranged^ 
m such away that after its conclusion a cause-effect relationship 
between the program or treatment and its observed outcoines qan be made. In 



o 



ther words » the evaluator must establish the Internal validity of the 



study. 



In order to do so, the evaluator must eliminate all other explanations 
or rival hypotheses for a change in the value of the dependent variable, 
other than the independent variable or in this case the program. Several 
commonly cited rival hypotheses are history, maturation, testing, instru- 
mentation, statistical regression, selection bias, and experimental mortality 
(Campbell and Stanley, 1972). History refers to the specific events occur-- 
ring between the first and second measurement in addition to the program or 
treatment. Maturation refers to processes within the respondents operating 
as function of the passage of time per se, including growing older, 
growing more tired, and the like. Testing refers to the effect of taking 

■ . t -3- . - - 
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a test upon the scores of n second tt*8tin^^. Tnst rumonr.at ion reftTt; to 
chnnfies In the calibration of the measuring tool or changes In tho 
observers used to collect the data. Statistical regression refers to 
.the value of scores on the measurement tool regressing toward the mean 
on a ]l^cond observation because the grolp has been selected on the basis 
of their extreme scores. Selection bias refers to biases resulting from 
differential selection of respondents for the comparison groups. Mortality 
refers to differential loss of respondents from the comparison group. 
Selertion-maturatlon interaction refers to different maturational rates 
of groups as a result of selection biases. 

To eliminate these rival hypotheses > the researcher may use a control 
group. A control group is a group like the treatment group In most ways 
except the absence of the treatment. Using a control group eliminates 
rival hypotheses or threats to internal validity as they are commonly 
referred to, in that the rival hypotheses should be manifested equally 
In both groups. If the program or treatment Is effective, post measures 
of the treatment group should be significantly different f^rom those 'of 
the control group. 

Th^ evaluator may also eliminate rival hypotheses through the use of ' 
randomisation; i.e., assigning subjects to the control or treatment group 
through a random procedure* Thus, randomization Insures that every subject 
to be assigned has an equal chance of appearing in. the treatment or control ^ 
group through a random pii)cedi>re. Randomization negates pre-existing 
differences between both groups* ' ^ 

If possible, the ^valuator should use both a.<;ontrol group and randomiza- 

• - „ ■ , ■ • ■ - I. 

tion procedure to eliminate rival hypotheses. A control group is not always 
sutficient to eliminate threats to internal vrflldity because the groins "^-^ 

_ ." -A- . - -0 . 
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may hi- different on chrtior varlnMos other tlian the troatraont. Usin^- 
ranHomizatlon only may not oltmliiat(^ such rival hypothost-s as toHtin^;, 
Btatistlcal regression » instrumentation, and hiBtory. Tf both techniqqoR 
are used* however, generally all thrtsits to Internal validity are 
eliminated. 

Including a control group in the study and using randomization are 
not the only conditions under which rival hypotheses are eliminated. If 
the Jgjj^reatment or program is of short duration, history and maturation can 
be eliminated as threats to internal validity without use of a control 
group. The rival hypothesis of testing ma/^lso be eliminated by the use 
of unobtrusive measures. If the treatment gtoup caH, be put in an extremely 
controlled environment.^ history can be eliminated as a rival hypothesis. 
When participation is mandatory, mortality is controlled for and when 
subjects are from the middle range of a distribution ^ regression as a 
threat to internal validity is controlled for. 

The evaluator must also give consideration to the external validity 
of the study. External validity addresses the question of generallzabllity 
to what populations^ settings, treatment variables^ and measurement variabl 
can the observed effect be generalized. Factors jeopardizing external 
validity are reactive effect.^ of testing, interaction effects of selection 
biases and the experimental variable, reactive effects of experimental 
arrangements, and multiple-treatment^ interference (Campb.ipll and Stanley,. 
1972^- The reactive effect of testing refers to a pre-test increasing or 
decreasing the respondents^ sensitivity or responslveiiless to the program or 
experimental variable, making results unrepresentative of the effect of 
the program or treatment variable on a population of respondents who^ have 



not been pr9~moa«urt»cl. . The Inlprnrt Ion crfc'crs of Krlcct lon nnd the 
exptTLniental^varlable rol-i^rs to the' I'.irt that the ohsr-rvf-d ^effect may bi^ 
a roHuit ot the unique charnctpriot Ics of the group selected. Reactive 
effectH of experimental arrangements refers to the respondents realizing 
they are in an experimental situation. Multiple- treatment interference 
refers to the situation where the same treatment is continually applied 
to the same respondents. The extent to which these threats to external 
validity can be eliminated is directly related to the extent one can 
generalize findings from one population of subjects to qnother. 

Evaluating the Program 

♦ 

The process of evaluating a program consists of the following 
steps: 

Step One: Identifying what is to be evaluated; 

Step 'ivo: Identifying what information is needed to conduct the 
evaluations; ' ^ , " 

Step Three: Selecting or developing the ins truments to collect 
the necessary information; 

Step Four; Defining success; and 

Step Five: Developing « management system. - 

Each of these step's will be further described in thVsf ollowlng 
sections. 

'^A. Step One: Identifying what is to be Evaluated. ^ 

identifying .what is to be evaluated is the most crucial step in 
the evaluation process; the degree of success In which^hia is carried 
o«t will limit the degree of comprehensiveness and utility the evalua- 

tion can attain. It is carried out by exaxiilning written documajjts 

^ ♦ 
concerning the program and meeting with the program director for the - 
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purpose of developing an explicit statement of wlint it Is the pro^'riim 
Intends to do. Simple na It seems, this 1b often n difficult task. 
When dealltig with newly dev<' loping prograras, oftt ii the program di- 
rector will he unsure, and sometimes rightly bo^ of what the goals 
of the program will be. In such tases the use of an advisory board 
consisting of designated "experts" from relevant fields or a review 
of the literature on other programs niay be feasible means of clarify- 
ing the Intentions of the partlculai/ program. In dealing with exlst- 
Ing programs » the program director may not see the need to develop 
a formal statement of program goals particularly If the program has 
existed and grown without a clear conception of Its goals. In other ^ 
cases, goals may exist but may be so general they are meaningless. 
However, by focusing, on what changes the program wishes to make In 
both knowledge and skill 1^ students, profession's working within 
the Instltution.and tje community in genei:^!. It is likely that^ 
some li^ppivant goals will emerge. 

Con?>letionA6f this Task is attained when both the evaluator 
and program director reach an agreement on what t^e program Intends 
to do and develop a formal statement specifying this. 

In summary. Step One. identifying what is to be evaluAted, may 
be broken down into the iiollowing tasks; 

1. Examine written material relevant to the program* ^ 

2. Meet with the program cji^^ctor to discuss the program 
goals. 

3. Develop a formal statement of what the program Intends to do. 
The first and second of these tasks need not progress in the 
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coinp lot c Si i»p Oiu». 

H. Stop Two: IdtMit I fy Inp. What i n t o t tna f 1 on Is N('<-cU»d to (:ou(!t\(:t ihe 
Rvaluntloti. 

After the evaluator and proxt^«iTi director have reached an agro.e^ 
C ment upon what the program Intends to do, they must next coroe to an 

agreement about what Information or data will document the successful 
attainment of the program goals. Thla process can be carried out by* 
the evaluator and program director Jointly developing a lint of several 
possibilities for each goal or by the evaluator developing a list of 
possibilities and subsequently presenting them to the program director. 
For example, if a goal of the program is to Increase student knowledge 
in accounting, successful attainment could be indicated by change in 
performance on an examination or by the students ability to correctly 
fill put a balance sheet. Frequently at this point it becomes evident 
^ that some goals need t<ij^be further clarified. Thus Steps One and 

'fwp may at times be approached concurrently. The focus at this point, 
however, is on Identification of the type of information that will 
document attainment of the goals; not on the particular Instrument or 
procedure to be utilized to collect the data* 

Once the list of types of Information that could document attain- 
ment of the goal Is developed, those to be utilized In thijr particular 
case can be selected by the evaluator. In making the selections, 
consideration must be given to reliability, validity, both internal 
and ea^emal and those practical matters previously addressed. 

In summary^ Step Two / identifying what information is needed to 
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Voiuhict tno fViiliuif Ion ,iMnis 1 Mt s of xhv following*, tnjikM: 

W I^.vt•^of. H?i^ rypt»M of Informarfon tlwjf 

would doriimont succi'ms f til iit t nf ntneT>t of t ht», ^^,u/il . 

2. St»loct t rom t>ho list dtwelopot! thoH%» most approprlnte. 

sivp Three: SelectlnK or DovelopliiK the Tn«trumontH to Collcc.r 

the Necessary Information 

Knowing what types of Information are needed, the, evaluator 

can then proceed to Ident Ify or develop specif Ic Inst rumen ts to 

collect the Informations Sources from which possible Instruments can 

be Identified include but are not limited to the following; 

1. Buro3» O.K. Mental Measurements Yearbooks . Highland Park, 
Gryphen Press 1938, 1940, 1«)49, 1953, 1959, 1965, 1972.; 

2* Buros, O.K. (Ed.) Tests In Print . Highland Park, NJ: Gryphen 
Press, 1961.; 

3. Test publisher catalogs; and 

A. Professional Journals and publications. 

I 

\ 

Consideration of data collection Instruments « however, should not 
be limited to standardized Instruments. Metfessel and Michael offer 
a variety of measures that can be utilized in evaluation of school 
programs in addition to standardized instruments (See Appendix). The 
final selection of inettuments and -techniques should be made in conslc|- 
eration of those guidelines of the American Psychological Association 
previously cited. 

If appropriate measures cannot be* Identified, the evaluator must 
qpnsider the feasibility of developing appropriate meacurefe. The deci- 
sion to do so or not will be influenced by the time, money and staff 
available for the evaluation^ Any instrument created must be developed 

-9- 
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rrVlt'wiMi with tlir ?;.ifm: itt'intlnv tluit cxlstiiir. i hn t riinir n t s arr. 

V 

to rolliu't tlw luvrSM/iry i n ! t)rin;it I on , prt)ctMHl*i by: 

» 

K I <!onl 1 fy InR poM^{lhl« oxlMtln>; inMtrumrntH ar t fchul qtieri 
aval lablc; 

2. Rt?vlewlng iho5u« inHLrumtMitH or toclmlques I clont I f loci; 

3- Select InK tkoHo mokt appropriate from those nvallable; and 

4. Developing InBt ruments , If feaHlble when appropriate^ 

Step Four: Deflnlnp, SucceHs 

After the «peclflc Instruments and technlquea have been oelectod, 
succesB must be operationally defined prior to beginning the data 
collection. This step is one whlcji must t>e left to the sole discre- 
tion of- the program director and program staff. For each instrument 

S ' 

or data collection technique a decision must.be made^ as to what 
results out of a^l of those possible will be consider-ed successful 
attainment of the goal. For example, on examloations a specific cut- 
off .score must be identified as distinguishing between successful and 
unsuccessful attainment of the program goal. If observation schedules 
are utilized, a specific number of occurrences of a given behavior 
may be agreed upon as indicating successful attainment of the goal. 
In many instances establl shment of a criterion level is difficult and 
anxiety producing for the program director, particularly If there is 
little prior data to base the decision on- If this Is the case, the 
evaluator should clarify to the program director that the arbitrariness 
of the decision will be addressed in the evaluation report; Neverthe- 
less, no evaluation data should be collected until these decisions 

-10- 
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are^mdde.v Without an, adequate understanding df what will bB con- 
sid^i^^ success between the program directar and evaluator, inter- 

pretatlop of ^ny data collected ^j||j^the "evaluation may^^ 

\ ' 
both parties feel are false accusations conceralng the success and 

failure of the* program. v , • . 

In sunraiary,^ Step Four, diffining success^ consists of one 

major task: ' . . - ' . 



1. Defining criterion levels of^uccess for all instruments* 
and data collection techniques by the program .director. 




E. Step Five: Developing of ^ Management System , 

. -■ . ' ' . 

Development of a, management system Is ^necessary to- Insure that 

all needed data is collected ^d synthesized^ to make it moa^meaningful. 

Developing a management system consists of making decision? in ^ 

regard to who .will distribute or administer the evaluation forms> how 

\ response to the forms will be enhanced; to whom the forms will be 

returned, the schedule of administration for the forms^ the types of 

analyses to be performed^ who will receive Immediate feedback of the 

V 2 
re^plts, and how the data will ultimately be synthesized. 

Making these d§.cisions can facilitated by'^ organizing the forms v. 

by whom the respondents will be. This process is likely to highlight 

instruments which are not essential to the collect;lon of the evaluation 

■ . ' '41 

data and .areas in^the instruments requiring further revision before 

. • . y . 

the Instruments are utilized. Thus, devel<ypment of the management 



"Forms" as used here refers to any document fcQm^which data are^collected 
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system can. serve" to clarify , further the focus and process of ^he ^ 
Investigation. 

In identifying' an. dndividiial to administer th^ forms i consldera-- 
t±on should' be g*lven to a person who* spends a conslderablevportlon of 
his/her time working with th^ progiram staff > is familiar with the forms 

■s. 

and whose request for cooperation will be seen as necessary ,to be in 
compliance with. . 

This individual should also typically be the Individual to whom 
the forms are returned to allow for the closest possible monitoring 
of the system. ;'Even djjQ^i^u^ual circumstances, where the information 
is urgently needed by the program developers It is unwise to have 

the forms returned directly to the program di^^rector. Program directors 

.t * ■ 

are frequently , involved in enormous amounts of administrative work 
and forwarding of th^ forms to the evaluation istaff is frequently 
given low priority iw Frequently, by routing , forms to the evaluation 
staff initially^ much of the administrators time can be save^. For 
example. Instead of reading thirty evaluation forms eacli of which 



minutes re«(dlhg summary report of the thirty forys forwarded to him 



require ten minutes, the administrator may only need to spend five 
llL 

by the evaluation staff. ^ 

Cajreful consideration must be given to what procedures will best 
Insure a high response rate. For some groups several follow-ups in 
the form of postcard reminders and telephone calls wAJ.! be necessary, 
•ilpr other groups use o£^ incentives, both monetary, materialistic and 
non-materialistic may be necessary. Sti^l, for other groups use of 
administration policies may be the best approacTi. - Familiarity with 
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the characteristics of the. respondents will facilitate selection 



of the most appropriate approach. In some cases, a combination of 
approadEies may be watrante^v 

The decision In regard to the schedule of administration for 

a 

the forms must be contingent upon the evaluation design and particular 
variables initiated in the evaluation. In addition to the evaluation 
design, consideration must also be given to the practicality of the 
situation and utility of the information collected. For example, it 
would be unw:^e to distribute surveys to teachers and expect a 
res-ponse during a week in which final examinations are given and grades 
are determined. Equally unwise would be the administration of an 
attitude scale midway through the academic program rather than at 
completion of the program when the aim is to determine attitude change 
after being in the program. Of course^ if the attitude srcale were 
adnninlstered. midway and at completion of the program, the midway 
measure would have some utility. 

Determining the types of analyses to be performed Is related 
to the objectives of the evaluation. There are three major modes of 
analyses; descriptive, correiatlonar and experimental. ^The dejscriptive 
approach Is self-explanatory, Its^ipurpose id to Identify what happened 
or what exists. The correlational approach also Includes the descriptive 



function, but goes beyond this irtlwetermining the extent to which two 
vArl^bles are related. The experimental approach* Includes the functions 
of the previous two approaches as well as^ Investigating causal- effect 
relationships. To select the appropriate mode of analysis the objectives 
of the evaluation must be matched with the function of the analysis 
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level. Associated with each level of analysis afe corresponding 
statistical techniques. Each statistical technique also makes 
ass%unptlons which must be met in order for the statistic to be 
validly used^ - ^ i ' . ^ 

Thus, the individual developing the plan of analysis must 
be competent in calculating the statlstics^^ and have adequate ^know- 
ledge of the assumption^ underlying the statistics* If someone on 
the program staff Is not ayailable^ an outside consultant should be 
recruited. 

Having made all the previous decisions, the final step* developing 
plans to synthesize the data that is to be col^lected must be made prior 
to installation of the system. The number of "synthesizing" documents 
should be kept to a minimum; howeyer, caution should be t|iken to avoid 
inclusion of data from documents that are unimportant and only 
peripherally related to each other. Typically, this step evolves 
into development of an outline for a final rieport of the evaluation^ ^ 
As this outline is developed, consideration must be given to whom the 
report is to be written, who will receive copies, and what the report 
will be used for. 

As the system is installed according to those policies originally 
agreed upon, it should be periodically reviewed to Identify changes 
indicated as necessary through application of the proposed system* 
Undoubtedly, some changes will be required due to the impossibility 
of foreseeing all outside restraints which will be encountered 
Influencing the data Collection. However, Tnatiagenient policies for 
administering the forms are essential from the beginning to serve as 
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guideposts for the Initial use of the forms* 
* ■ ' * ^ 

In summary. Step Five^ developing a management system can be 

broken down Into the following tasks:, 

f 1. Deciding who will respond to which forms; 

/ 2. Deciding who will distribute the forms; 

/ 3/ Deciding to whom the forms will be returned; 

r 4« JDesigning a schedule for administration of the forms; 

/ ■ . ■ 

^ '5. Developing a plan of analysis; 

6. Identifying who \>ill receive immediate feedback; and 

7. Identifying how the data will be synthesized. 

In retrospect what the preceeding steps in conducting a program 

• * ■* 

evaluation have involved are development of a plan and adherence to 

III. Evaluatlrig the Vroject 

Project evaluation is generally carried out in the initial stages 
of program development. As stated previously proj[ect evaluatl^ is 
used to provide means of defining and monitoring program progress; to 

assure identification of problems before they disrupt activities in 

' ' - V » ■ ^ 

other parts of the program; to maintain a record of WMt is in fact dorie 

and why;' and to provide data for decision- making* After the program is 

developed and stabilized, project evaluation may not be necessary. 

The process of evaluating a project consists of the following steps; 

Step Orii^: Specifying activities ♦and criteria for success^ful 
^ ' \ performance^ 

Step Two: Monitoring progress towards completion of the activitie 



^tep Three: Documenting' completion of the activities; and 

Step Four: Evaluating the completed activity. ^ 

Prior to initiating project evaluation it is essential that a well 
thought*-out work plan be deVifcioped, The work plan should identify the 
major activities which must be accomplished to complete the project and 
thelT relationship to each other. Program Evaluation Review Techniques 
(PERT) should then be utilized to put the activities Into a time frame* 
The staff of the project should participate as much as possible In develop- 
ment of the work plan to develop a sense of ownership and willingness to 
cooperate in completing tasks assigned-^ Folldwing* this the evaluation cjan 
then begin. In a project with a relatively small staff the steps lij pro- 
ject evaluation can be * carried out on an Informal basis. In a project 
with a relatively large staff it may be necessary to develop a forma 
mechanism to c^arry these activities out. The subsequent sections present 
an example of a formal mechanism for conducting the evaluation*- Specific ' 
projects may wish to adap^t these to their own needs for formal or informal 
use. 

A. Step One: Specifying Activities and Criteria for Successful Performance 
Taking the work plan, each activity can be broken down into a 
series of tasks to be assigned' to a project member. The Task Speci- 
fication Form in Figure 1 or one similar to it can be utilized by the 
project director to assign the task. ^ , , 




Useful references explaining PERT include the following: 

Case, CM. "The application of PERT to Large-Scale Educational and Evalua- 
tion Studies." Educational Technology , Oct. 1969, 9, pp. 79-83, 

Cook, D*L. Program Evaluation and Review Technique Applications in Educa - 
tion . Washington, D.C. : U.S. Government Printing Office, 1966. 

Special Projects Office, Department of the Navy, PERT Summary Report , 
Phase I. Washington, D.C: U.S. Government Printing Office, 1962* 



TASK^ SPECIFtCATTOM F0R:I 



Task JIuaber: 
Assi^ed to: 



To be conpleCed by: 



Objective task is relatecf "to in overall plan: 



Task Description 



(Describe the task in eaough detail^ that it is replicable.) 



p.»....r rnrnnletion C^terU (Specify the fonn in «hich data ^.^^f 

and the char^cterxstxcs of the data vtixch wiix 

acceptable.) ; 



i 



Process donpletion Criteria (Specify parties to be involved, the flpw of 
Process opnpietxo feedback from the task and decisions to be made.) 
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The coBipletion criteria included on the form should be of 
two typepc product ahd process. The product comp-letlon criteria 
should be a description of what the observable end product of 'the 
task should Include. It should specify the form in which data 
are to be t>resented and the characteristics of the data which will 
be acceptab'le. The process completion criteria should specify parties 
be involved, the flow of feedback from the task, and decisions to 
/be made. The process completion criteria should describe the task 
^rformance in enough detail that it is repllcable and it should 
uphold all prior decisions that were made. - 

The value of this form lies in developing clear understioidlng 
between the task doer and the task assigner of what it is that is 
to be done. 

B. Step Two: Monitoring Progress Towards Completion of the Activities 
If the activities are put into a time frame as the work plan 
is developed, various times can be identified for making progress 
checks on the completion of each task. Thus problems can be identified 
as they emerge and alternative strategies developed for completing ^ 
the task, altering the task or id^*n4ifylng new tasks. The form 
presented in Figure 2 is useful for this type of monitoring. The form 
presented in Figure 3 is useful for documenting revisions in tasks or 
strategies. i . 

Thei Task Progress Report Form in TlgliTe 2 should be released by 
the evaliiatoi; to the task doer and returned by the task doer to the 
evaluator. The Task Modification Form should be completed by the 
evaliiator in cooperation with the program director and task doer. - 
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task' PaOCRESS, REPORT FORM 



Task Number 
Assigned 'to 



Original Gonpletion Dat^: 



Vmat steps have ^...L^ thus, far to perfdr. the tasK. 

/ . 



Estimate 
tion date. 



.Ua llklihood of^our collating the, task o. the specified ccple- 



1. I definitely will not complete the task. 

2. I ao uccertam 1£ I will »e ahle to complete th^ task. 



3. I definitely 
date. 



''will complete the task on the specified completion 



1 o-r 9 ' indicate specific problems yo 
If you have checked responses 1 or 2, indicate p 

encountering. 



/ 



kl are 
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Figure 3 



TASK MODIFICATION FORM 



Task Number: 



Today's Date: \ 



Assigned to: ' - 

Original Completion Date: ^ 



-r- 



Check the areas in which changes have been made in regard to thd Task 

Specification Forta . ( - 

I 

Completion Date ^ ^ i 



Person Assigned the Task A 



Product Criceria 
Process Criteria 



Indicate specifically any changes that have been made and the reason for 
the change. 



EKLC 



\ 
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H C. JStep Three: Documenting Completion 6f thty^ctlvltles 

As each task is comp-ieted documentation should be obtained from ^ 
the task doer in regard to both product and process. The Task Complex 
tion Form in ^!^ure 4 can assist in this documentation. This form 
should be completed by the task doer. 

D. Step Four: Evaluating the Completed Actlvl^ty 

After the task is completed, the evaluator ,8hould judge whether 
or not the product and process completion criteria have been met. 
(The Task Evaluation Form in Figure 5 can be utilized for this.) 

"if 

If the criteria have not been met, a decisdon must be made In regard 



to ttie appropriate technique for proceeding. 

In retrospect, this approach to project evaluation follows the 
Discrepancy Evaluation Model for . evaluation proposed by Malcolm' 
Provus ^"^^^^^ The model defines program standards aa criteria for 
each program stage; determines whether a discrepancy exists between 
some aspect of program performance and the criteria governing that 
stage; and uses the discrepancy information either, to change perform 
mance or to change program standards. When discrepancies have been 
identified, the next sequence of steps in the model are to determine 
where there is a discrepancy; what cor^ctlve actions are possible; 
and what corrective action is best. At each stage there are four 
alternatives for program activities: (1) go on to the next stage, 
(2) recycle the stage after there has beej^^ change in the program's 
standards or operations, (3) recycle to an earlier stage, or (4) termi- 
nate the project. The model has been particularly helpful in checking 
out pilot or new programs; diagnosing programs that are not functioning 
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;CASK^-€OMrLETI0M FORM 



Task NuDiber: 



Coapleted by: 



/ Completion Date: 



^ Product Completloa 



(Present or attach a cppy of the product completion 
data previously specifi^ed in the Task. Specification 
Form. ) 



Process Conpletlon 

Villa t other parties were involved? 



To whom was feedback of the task reported to : 



Were nev decisions Eiade? 



FRir 
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' 'TASK E^\\LUATIOM FO?J! 



Task Number: 
Coinpleted by: 



Conpletion Date; 
Today's Date: ^ 



Product Coapletion Criteria 

Does the product completion data presented in the Task Completion Form meet 
the product completion criteria pre<?iously established for the task? If 
not, indicate specific discrepancies that exist. 



Process Completion C^tc^rxa. 

Were all appropriate parties involved? If not. indicate who was oaiitted, 



-7 



Did the process uphold all prior decisions that were made? If not, indicate 
which decisions were violated? 



Was the feedback of the process reported to the appropriate parties? If 
ngt, indicate who was neglected. 



If new decisions were made, list future project objecti^/es you feel they 
will effect. 
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well;" and in Improving program efficiency. Thus, t.h^ model appears 
to be a usetul fraroevjork around which to develop the project evaJLuatl 
for program development. '•^Ti 




When utilized on a formal basis the proposed system may be likely 
to create more paper work than the benefits accrue^d from utilizing it* 
If this is the case it is recommended that forms be combined or 
eliminated to fit the particular situation. 



An evaluator may be one of two tjrpes; a project evaluator or a program 
evaluator. The project evaluator' s services are utilized In the Initial 
stages of program development. To function adequately the project evalua- 
tor should be highly familiar with the project; he should be a member of 
the project team. In essence he should serve as a manager of activities. 
The program evaluator' s services are utilized as the program develops ani^ 
throughout the life of the program* To function adequately the program 
evaluator should be familiar with the program but he should not be a 



full-time mexhber of the program staff least his objectivity and credibility 
be diminished. In essence he should serve as a program planning consultant. 
Because of the conflicting need for the project evaluator to be highly 
familiar with the activities of the project and for the program evaluator 
to be familiar yet distant enough from the program to maintain objectivity, 
it is preferred practice not to Intermix the two roles. 



IV. Summary 
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*Multlple Criterion Measures for 
Evaluation of School Programs 
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Pages 25-9 were removed from 
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*Thls list has been developed by Newton Metfessel and William 
Michael and appears in the following source: 

Worthem, Blaine, and Sanders, James. Educational Evaluation ; Theory 
and Practice . Worthington, Ohio: Charles A. Jones Publishing Coicpany-,* 1973, 
pp. 274-279. 
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